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with  several  problems  in  the  area  of  digital  communications  and  the 
elimination  of  redundancy.  The  importance  of  the  problem  arises  from  the 
growing  use  of  digital  transmission  in  the  military  as  well  as  the  civilian 
communications  field.  There  are  many  advantages  enjoyed  by  digital 
transmission,  the  most  notable  being  that  there  is  almost  no  signal  to  noise 
degradation  when  relayed  through  a number  of  repeaters,  whereas  analog 
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20.  Abstract 


repeaters  lose  3 db  every  time  the  number  of  repeaters  is  doubled.  The  one 
marked  disadvantage  of  digital  transmissions  is  that  digitization  of  a 
basically  analog  source  such  as  speech  or  TV  results  in  a bandwidth 
expansion  using  conventional  techniques.  Good  quality  digitized  voice 
requires  on  the  order  of  50,000  bits  per  second  if  data  compression  is 
not  used.  With  conventional  techniques,  this  requires  30  kHz  of  bandwidth 
while  the  analog  voice  signal  could  be  transmitted  over  a 3 kHz  channel. 

The  area  of  bandwidth  compression  seeks  to  remove  this  disadvantage  by 
e iminating  the  redundancy  in  the  signal.  Over  30  papers  and  reports 
were  published  on  this  subject  by  the  PI'Wduring  this  period. 
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SECTION  I 
INTRODUCTION 

Both  military  and  civilian  communication  systems  are  becoming 
increasingly  digital,  as  opposed  to  analog,  in  nature  due  to  several 
factors: 

1.  Digital  transmission  can  be  made  more  noise-immune  through  the 
use  of  error  correcting  and  detecting  codes.  This  applies  to 
both  natural  and  malicious  noise  (e.g.,  jamming). 

2.  The  cost  of  digital  hardware  has  decreased  markedly,  and  there 
is  every  indication  the  trend  will  continue. 

3.  Digital  transmission  suffers  almost  no  signal-to-noise  degrada- 
tion when  relayed  through  a number  of  repeaters.  Analog 
repeaters  lose  3 dB  every  time  the  number  of  repeaters  is 

doubled.  Thus,  for  example,  eight  repeaters  cause  a 9 dB  loss 

-5 

in  an  analog  system.  On  a digital  system  operating  at  a 10 
bit  error  rate  the  corresponding  loss  is  only  1 dB. 

4.  Digital  transmission  is  much  more  amenable  to  protection  by 
cryptographic  techniques. 

The  one  marked  disadvantage  of  digital  transmission  is  that  digitiza- 
tion of  a basically  analog  source  such  as  speech  or  TV  results  in  a band- 
width expansion  using  conventional  techniques.  Good  quality  digitized 
voice  requires  on  the  order  of  50,000  bps  if  source  coding  is  not  used. 
With  usual  modulation  techniques  this  requires  a 25  to  50  kHz  channel, 
whereas  the  analog  voice  signal  could  be  transmitted  over  a 3 kHz  channel. 
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For  a given  bandwidth  allocation  the  digital  system  will  thus  be  able  to 
handle  only  about  one-tenth  the  traffic  that  could  be  handled  by  an 
analog  system. 

Source  coding  (also  known  as  bandwidth  compression  or  data  reduction) 
attempts  to  remove  this  disadvantage  by  reducing  the  required  data  rate. 
For  example,  source  coding  of  voice  with  a ten-to-one  compression  factor 
would  allow  as  many  digital  as  analog  channels.  Source  coding  provides 
the  additional  advantage  of  reducing  the  signal  energy  required  to 
transmit  a block  of  data. 

Although  source  coding  holds  promise  for  the  use  of  digital  trans- 
mission, cost  is  a major  obstacle  to  its  use.  Source  coders  which 
achieve  significant  compression  have  been  complex  and  therefore  expen- 
sive. Another  aspect  of  this  complexity  problem  has  further  hindered 
the  use  of  source  coding  in  remote  telemetry  and  other  systems  where 
operations  performed  at  the  transmitter  are  orders  of  magnitude  more 
expensive  than  operations  performed  at  the  receiver.  This  has  to  do 
with  the  fact  that,  to  date,  the  source  encoder  has  borne  the  brunt  of 
the  overall  system  complexity,  while  the  source  decoder  has  been  much 
simpler.  This  is  in  contrast  with  channel  coding  (coding  for  error 
correction)  where  the  channel  decoder  must  be  complex,  while  the  channel 
encoder  can  be  simple.  This  partially  explains  why  channel  coding  has 
found  more  widespread  use  than  source  coding.  Besides  remote  telemetry 
(e.g.,  space,  ocean,  behind  enemy  lines),  systems  which  have  a large 
ratio  of  transmitters  to  receivers,  or  in  which  the  transmitter  is 
expendable,  will  have  an  asymmetric  complexity  cost,  with  the  transmitter 
complexity  having  the  higher  cost.  Examples  of  such  systems  are  RPV's 
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SECTION  II  - A 

CONVOLUTIONAL  SOURCE  ENCODING 


Most  source  coding  schemes  utilize  complex  encoders  and  are  thus 
not  well  matched  to  remote  telemetry  and  other  applications  where  the 
encoder  must  be  simple,  inexpensive  and  reliable.  We  have  developed  a 
convolutional  source  encoding  technique  which  is  well  matched  to  such 
applications  because  the  encoder  is  extremely  simple.  ["Convolutional 
Source  Encoding",  IEEE  Trans,  on  Info.  Theory,  Nov.  1975.] 

The  decoder  is  much  more  complex,  and  effort  was  devoted  to 
simplifying  the  decoder.  A small  simulation  on  video  data  indicates 
that  compression  ratios  of  2:1  or  more  are  possible  without  introducing 
any  distortion  into  the  reproduced  data  (noiseless  coding). 

The  basic  system  utilizes  a convolutional  encoder  and  a sequential 
decoder.  The  encoder  operates  at  rates  greater  than  one,  whereas  con- 
volutional coders  used  for  channel  coding  operate  at  rates  less  than  one. 

It  is  easiest  to  understand  the  operation  of  our  convolutional 
source  coding  system  by  first  understanding  its  operation  when  used  for 
joint  source  and  channel  coding.  In  usual  systems  the  data  is  encoded 
in  two  stages.  First  a source  coder  removes  redundancy  from  the  data 
to  reduce  the  number  of  bits  which  must  be  communicated.  Then  a channel 
coder  adds  redundant  bits  in  a controlled  manner  so  that  most  errors 
which  occur  on  the  channel  can  be  corrected.  If  source  coding  removes 
redundancy  and  channel  coding  adds  it  back  in,  one  might  try  to  dispense 
with  both  operations  and  use  the  natural  redundancy  of  the  source  for 
error  correction.  A simple  example  shows  why  this  cannot  usually  be 
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done.  If  the  message  "I  AM  NOT  ABLE  TO  PROVIDE  SUPPORT."  is  trans- 
mitted directly  and  a single  character  is  received  in  error:  "I  AM  NOW 
ABLE  TO  PROVIDE  SUPPORT."  is  received.  Not  only  is  the  error  over- 
looked, but  the  meaning  of  the  message  has  been  reversed.  This  is  in 
spite  of  the  fact  that  the  amount  of  redundancy  is  more  than  sufficient 
to  correct  single  errors.  Although  the  redundancy  is  large  it  is  not 
evenly  distributed. 

We  have  developed  a very  simple  device  to  transform  the  redundancy 
of  the  source  into  a usable  form.  This  joint  source  and  channel  en- 
coder is  a rate  one  convolutional  encoder.  It  consists  of  a shift 
register  and  a number  of  mod-2  adders  (exclusive  OR  gates).  A typical 
encoder  is  shown  below: 


input  bits 

> 

from 

source 


mm 

i 

m 

M 

m 

mm 

■i 

1 

bits  to 
channel 


This  encoder  has  sixteen  stages,  and  typically  25-100  stages  will  be  more 
than  adequate.  The  output  bits  are  transmitted  over  the  noisy  channel 
and  placed  through  the  decoder  which  performs  the  inverse  operation  of 
the  encoder.  The  decoder  which  corresponds  to  the  above  encoder  is 
shown  below. 
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It  is  seen  that  because  of  the  feedback  in  the  decoder,  even  a single 
transmission  error  will  cause  many  errors  in  the  decoded  information. 

If  there  are  enough  stages  in  the  encoder  then  the  output  following  an 
error  is  complete  gibberish.  For  example,  the  same  single  character 
error  as  in  the  message  above  causes  the  output:  "I  AM  NOWJ.NXAAVWM.E 
WTYROVBGZ,  RI".  It  is  easily  seen  that  the  characters  most  likely  to 
be  in  error  are  the  W and  the  J of  NOWJ...  . A first  attempt  would 
be  to  change  the  J of  NOWJ...  to  a blank,  making  the  text  "I  AM  NOW...". 
This  puts  two  errors  in  the  decoder,  the  first  being  due  to  the  channel, 

the  second  being  due  to  our  attempted  correction.  Thus  the  output  is 

again  gibberish:  "I  AM  NOW  HU.CVKIWXRORBHUWTZHUIGK*" . When  we  try 
correcting  the  W we  find  the  output  is  meaningless  except  when  the 
proper  correction  to  a T is  tried.  This  example  used  a 5 bit  code 

with  star  = 00000,  A = 00001,  ...  Z = 11010,  blank  = 11011,  period  = 11100, 

comma  = 11101,  quote  = 11110,  question  mark  = 11111.  Low  order  bits 
are  encoded  first.  This  code  has  lower  redundancy  than  most  codes  in 
common  use  (e.g.,  ASCII)  and  therefore  we  expect  even  greater  correction 
capabilities  with  these  other  codes. 

Note  the  simplicity  of  both  the  encoder  and  the  feedback  decoder, 
each  consisting  of  a shift  register  and  several  exclusive  OR  gates.  The 


11-3 


only  complexity  involved  is  in  recognizing  what  is  and  what  is  not  a 
meaningful  message  and  in  deciding  whi'-h  errors  are  most  likely.  It  is 
the  necessity  of  having  such  a capability  which  makes  the  receiver  more 
complex  than  the  transmitter.  This  requirement  can  be  met  by  having  a 
sequential  decoder  available  at  the  receiver.  The  sequential  decoder 
need  not  have  a complete  description  of  the  source  characteristics,  but 
of  course  a less  complete  description  doesn't  allow  as  great  an  error 
correction  capability. 

The  sequential  decoder  is  similar  to  usual  sequential  decoders 
except  that  its  metric  has  an  added  term  which  depends  on  the  source 
statistics.  For  a discrete  memoryless  source  with  probaDility  distri- 
bution q(u)  on  its  outputs,  and  a discrete  memoryless  channel  with  trans- 
ition probabilities  p(y|x),  the  metric  increment  on  a branch  is 

m = In  + In  q(u)  . 

We  have  shown  that  so  long  as  H,  the  entropy  or  rate  of  the  source,  is 
less  than  C,  the  capacity  of  the  channel,  reliable  transmission  is 
possible.  That  is,  as  the  constraint  length  of  the  convolutional  encoder 
v tends  to  infinity  the  probability  of  a decoding  error  tends  to  zero. 
This  one  step,  combined  source  and  channel  encoder  is  thus  as  good  as 
the  best  two  step,  source  and  channel  encoders. 

If  H > C we  must  go  to  a lower  rate  convolutional  code  to  obtain 
reliable  transmission.  If  H < 2C  then  a rate  one-half  code  suffices;  if 
H < 1.5C  then  a rate  two-thirds  code  suffices;  etc. 

Conversely,  if  H < C it  would  seem  there  should  be  some  margin  for 
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using  a higher  rate  code.  In  the  extreme,  with  a noiseles1"  channel  there 
is  no  point  in  encoding  the  data  with  our  rate  one  encoder  since  no  error 
protection  is  needed.  The  question  is  how  to  generate  information  loss- 
less convolutional  codes  with  rates  greater  than  one.  Somewhat  surprisingly, 
if  H < C/2  then  just  throwing  away  every  other  output  from  our  rate  one 
encoder  results  in  a rate  two  encoder  which  allows  reliable  decoding  using 
the  previously  described  metric.  If  H < 2C/3  then  throwing  away  every 
third  output  from  a rate  one  encoder  works,  or  we  can  take  a rate  one- 
half  encoder  and  throw  away  the  1st,  2nd,  4th,  5th,  7th,  8th,  etc.  set 
of  outputs.’  Either  technique  yields  a rate  three-halves  encoder.  A 
little  thought  shows  that  these  encoders  are  the  obvious  generalization 
of  error  correcting  convolutional  encoders  to  rates  greater  than  one. 

It  is  interesting  to  note  that  the  encoder  is  universal  (e.g.,  in 
the  English  language  example  the  same  encoder  could  be  used  to  encode 
German).  Only  the  decoder's  metric  depends  on  the  source.  Again  this 
is  well  suited  to  applications  in  which  encoder  complexity  must  be  low, 
since  frequently  the  source  statistics  are  not  known  a priori  and  a non- 
universal  encoder  would  have  to  be  programmable.  With  our  system  the 
first  few  messages  (e.g.  photographs  of  a planet)  could  be  repeated  to 
allow  decoding  without  knowledge  of  the  source  statistics.  The  source 
statistics  could  be  estimated  and  used  on  future  messages  which  then 
need  not  be  repeated. 

Another  universal  approach  is  to  compile  statistics  on  the  source 
while  encoding  its  output.  These  statistics  are  transmitted  at  the  end 
of  the  message  and  allow  use  of  the  optimal  metric  at  the  decoder.  Be- 
cause our  encoder  is  universal,  a 2:1  compression  is  possible  using  the 


same  encoder,  no  matter  what  the  statistics  of  the  source  are,  provided 
they  allow  a 2:1  compression  (i.e.,  the  source  is  at  least  50%  redundant). 
Because  of  the  encoder's  universality  the  source  output  need  not  be 
stored  for  encoding  after' the  statistics  are  known.  Rather,  the  compressive 
encoding  proceeds  while  the  statistics  are  compiled.  Of  course,  the  de- 
coder must  store  the  received  data  until  the  statistics  are  transmitted. 

But  in  applications  where  decoder  complexity  can  be  many  times  greater 
than  encoder  complexity  (e.g.,  remote  telemetry)  this  is  no  problem. 

The  proof  that  reliable  decoding  is  possible  uses  arguments  from 
the  theory  of  branching  processes.  It  does  not  treat  the  problem  of 
computational  effort  and  as  one  might  expect  there  is  a problem  analogous 
to  the  existence  of  the  computational  cutoff  rate  for  normal  sequential 
decoding.  Koshelev  independently  realized  the  applicability  of  con- 
volutional codes  to  one  step  encoding  and  derived  bounds  on  the  ex- 
pected number  of  computations.  He  shows  that  a source  has  a computational 
cutoff  entropy  Hcomp  which  is  larger  than  H and  that  the  expected  number 

of  computations  per  bit  is  finite  only  when  H <R 

r comp  comp 

We  also  investigated  application  of  these  techniques  to  real 
facsimile  data.  Knowing  the  behavior  of  the  algorithm  on  such  sources 
should  allow  easier  extensions  to  other  sources.  We  concentrated  on 
facsimile  for  several  reasons. 

1.  It  is  an  important  area  for  source  coding.  A high  quality 
photo  requires  on  the  order  of  107  bits  for  digital  trans- 
mission before  source  coding.  For  satellites,  space  probes 
and  photo  reconnaisance  where  hundreds  of  photos  are  in- 
volved this  represents  a tremendous  amount  of  information. 

Source  coding  has  been  used  successfully  in  some  instances, 
but  simpler  encoders  would  be  cheaper  and  more  reliable  and 
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would  therefore  find  more  widespread  use. 

2.  Solving  the  problem  of  source  coding  for  facsimile  is 
probably  a necessary  first  step  toward  solving  the  problem 
of  source  coding  for  television.  The  video  portion  of 
television  is  just  a sequence  of  related  facsimile  pictures. 

3.  By  first  dealing  with  black/white  (no  gray)  facsimile  we 

can  simplify  the  problem  and  concentrate  on  the  basic  problem 
of  source  coding  a binary  source  with  memory. 

A preliminary  investigation  brought  several  problems  to  light. 

The  most  pressing  problem  was  to  find  a good  metric  for  facsimile.  In- 
tuition and  trial  and  error  did  not  seem  to  be  fruitful  approaches. 
Analytical  studies  indicated  how  the  metric  should  depend  on  the  source 
statistics,  and  we  compiled  and  used  such  statistics  in  the  design  of 
our  metric. 

Another  problem  studied  concerned  high  detail  areas  of  the  picture 
which  overload  the  computational  abilities  of  the  decoder,  while  low 
detail  areas  waste  transmission  time.  We  studied  the  use  of  a variable 
rate  encoder  in  which  the  transmission-starved  high  detail  areas  can 
effectively  borrow  transmission  time  from  the  transmission-rich  low 
detail  areas.  This  approach  was  used  in  our  facsimile  simulation. 


Tree  and  Trellis  Codes 

The  basic  Shannon-style  source  coding  theorems  relating  the  optimum 
achievable  average  distortion  for  a data  compression  system  to  the  dis- 
tortion-rate function  were  developed  for  time  invariant  trellis  or  tree 
coding  systems  and  ergodic  sources.  Tree  codes  consist  of  a time-invariant 
possibly  nonlinear  filter  (a  sliding-block  code)  as  decoder  and  a matched 
tree  search  algorithm  such  as  the  Viterbi  algorithm  which  finds  for  a 
given  source  sequence  a channel  sequence  which  will  drive  the  decoding 


filter  so  as  to  produce  a good  reproduction  to  the  original  sequence. 
These  results  removed  the  requirement  of  previous  coding  theorems  for 
either  time-varying  decoders  or  infinite  length  decoders  and  hence 
better  modeled  practical  data  compression  systems.  The  basic  coding 
theorems  were  also  extended  to  systems  where  the  source  is  incorrectly 
or  incompletely  specified  (universal  coding).  These  results  were 
published  in  [1,2]. 

The  above  mentioned  results  guarantee  the  existence  of  good  tree 
coding  data  compression  systems.  As  is  often  the  case  with  the  Shannon 
theory,  however,  the  theorems  are  existence  proofs  and  provide  no  in- 
dication of  how  to  actually  design  a data  compression  system  for  a 
given  source  and  distortion  measure.  It  was  shown  in  the  above  research, 
however,  that  the  problem  of  designing  a good  decoder  (which  thru 
specifies  the  encoder)  is  equivalent  to  designing  a filter  such  that 
when  driven  by  a memoryless  discrete  sequence  it  produces  a good  "fake" 
or  simulation  of  the  original  source.  This  led  to  the  development  of 


the  fake  process  approach  to  the  design  of  tree  coding  data  compression 
systems.  Roughly  speaking,  several  intuitive  techniques  were  used  to 
produce  good  fakes  of  given  source  models  by  matching  the  marginal 
probabilities  and  the  second  order  averages  (autocorrelation).  These 
systems  were  developed  and  simulated  for  Gaussian  memoryless,  auto- 
regressive, and  moving  average  processes  with  a mean-squared  error 
fidelity  criterion  and  one  bit  per  symbol  compression  systems.  These 
systems  worked  quite  well  providing,  for  example,  an  improvement  of 
.7dB  over  Lloyd-Max  optimal  quantization  for  memoryless  sources,  1 dB 
over  predictive  quantization  on  autoregressive  sources,  and  over  5 dB 
over  predictive  quantization  on  moving  average  sources.  This  improve- 
ment was  effectively  half  the  distance  between  the  traditional  systems 
and  the  optimum  achievable  as  given  by  the  Shannon  distortion-rate  curve 
and  it  was  obtained  at  the  expense  of  only  a moderate’ increase  in 
complexity.  This  research  is  reported  in  detail  in  [3,4]. 

Speech  Compression 

Our  initial  work  on  speech  compression  was  the  adaption  of  the 
traditional  theory  of  the  asymptotic  behavior  of  optimum  quantizers 
and  the  Lloyd-Max  algorithm  for  computing  optimum  quantizers  to  the 
problem  of  single-symbol  quantization  of  the  reflection  coefficients 
arising  in  Linear  Predictive  Coded  (LPC)  speech  compression  systems. 

The  principle  novelty  here  was  the  use  of  the  log  spectral  distortion 
measure  popular  in  the  speech  community.  This  distortion  measure  is  not 

of  the  usual  form  assumed  in  information  theory  and  it  required  a new 
app/oach.  We  developed  experimental  rate  vs.  distortion  curves  for  uniform 
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quantization,  uniform  sensitivity  quantization,  equal  area  (maximum  en- 
tropy) quantization,  and  minimum  deviation  quantization  and  the  relative 
merits  of  the  systems  were  compared  and  contrasted  for  fixed  and  variable 
rate  systems.  The  principle  conclusions  of  this  work  were  somewhat 
negative,  but  not  surprising  for  an  information  theorist.  It  was  found 
that  when  one  is  constrained  to  single-symbol  quantization,  i.e.,  to 
separately  quantizing  individual  reflection  coefficients  and  hence  be 
unable  to  use  the  correlation  among  these  quantities,  then  optimal  and 
suboptimal  schemes  provide  nearly  the  same  performance  and  hence  one 
might  as  well  use  the  simplest  scheme.  Thus  if  one  wishes  to  obtain  any 
real  improvement  over  the  LPC  systems,  it  is  necessary  either  to  use 
the  correlation  of  the  reflection  coefficients  within  a time  frame  or 
sequentially  in  time.  This  work  was  published  in  [5]. 

The  above  research  led  us  to  begin  to  look  at  ways  of  more 
efficiently  compressing  the  reflection  coefficients  as  well  as  alternatives 
to  the  LPC  approach.  Two  possible  approaches  were  suggested  by  our  pre- 
vious work: 

1.  Develop  a Lloyd-Max  style  algorithm  to  optimally  quantize 
the  entire  vector  of  10-12  reflection  coefficients  for  each 
time  frame. 

2.  Develop  a fake  process  tree  code  for  direct  application  to 
the  original  waveform.  These  approaches  took  a long  time 
to  implement  because  of  two  fundamental  problems: 

1.  It  was  not  clear  what  mathematically  tractable  yet 
subjectively  meaningful  distortion  rcsure  could  be 
used  to  define  "optimal",  and 


11-10 


2.  There  do  not  exist  good  generally  accepted  probabilistic 
models  for  speech  and  hence  techniques  like  Lloyd-Max 


which  require  a model  could  not  be  used. 

The  first  problem  led  to  a large  literature  search  and  many  discussions 
with  people  in  the  speech  field.  We  found  useful  characterizations  of 
many  proposed  distortion  measures,  developed  their  mathematical  properties 
and  their  relative  strengths  and  weaknesses,  and  we  summarized  our  find- 
ings in  a preliminary  Technical  Report  [6]  so  as  to  make  this  yet  un- 
polished work  available  to  interested  people.  Our  main  conclusions 
were  that  for  several  reasons  the  Itakura-Saito  distortion,  the  log 
spectral  deviation,  and  the  filter  distortion  were  the  most  promising. 

On  the  probabilistic  model  problem,  we  developed  an  algorithm  (described 
in  the  subsection  on  Quantization)  that  will  take  a quantizer  and  a data 
sequence  (a  training  sequence)  and  will  produce  a locally  optimum 
quantizer  by  iteratively  improving  the  partition  and  the  reproduction 
values  that  define  the  quantizer.  An  initial  quantizer  for  a vector  of 
reflection  coefficients  is  chosen  by  a technique  resembling  a pre- 
dictive quantizer  across  the  reflection  coefficient  vector  and  then  the 
iterative  algorithm  is  applied.  The  training  sequence  is  simply  LPC 
coded  standard  speech  files.  Numerous  simulations  and  the  results  look 
good  on  paper  (comparable  signal  to  noise  ratios  in  terms  of  spectral 
deviation  at  half  the  usual  LPC  rate),  but  we  are  awaiting  digital-to- 
analog  conversion  (LPC  synthesis)  which  must  be  done  elsewhere  before 
making  any  specific  claims.  This  research  will  be  continued  under 
AFOSR  Contract  #F49620-78-C-0087.  We  also  feel  that  if  this  research 
proves  successful,  it  will  provide  the  needed  codebook  of  elementary 


speech  sounds  to  use  in  a direct  tree  coding  system  not  involving  any 
on-line  LPC  analysis. 

Optimum  Quantization 


I As  an  offshoot  of  the  above  work  on  quantization  of  reflection  co- 

efficients a new  and  simple  development  of  the  basic  theory  of 
asymptotically  optimum  quantizers  was  obtained.  In  particular,  it  was 
shown  that  the  basic  properties  of  asymptotically  optimum  quantizers 
follow  easily  from  standard  inequalities  of  information  theory  (such 
as  Jensen's  inequality)  and  that  variational  techniques  are  not  re- 
quired. This  work  was  published  in  [7]. 

The  classical  Lloyd-Max  algorithm  for  determining  an  optimum 
quantizer  for  a specific  source  and  distortion  measure  requires  a 
probabilistic  description  of  the  source.  In  practical  situations, 
however,  this  model  must  be  obtained  by  observing  a "training  sequence" 
of  data  and  applying  statistical  techniques.  We  developed  a simple 
and  convergent  algorithm  that  operates  directly  on  the  data  without 
any  explicit  model  construction  and  produces  a locally  optimum 
quantizer  in  a sample  average  sense.  If  the  source  is  ergodic,  then 
our  quantizer  will  converge  to  the  Lloyd-Max  quantizer  for  the  "true" 
underlying  probabilistic  model  in  the  limit  of  a long  training  sequence. 
Our  technique  is  a simple  sequence  of  optimizing  a partition  for  a set 
of  reproduction  values  and  then  optimizing  the  reproduction  values  for 
the  given  partition.  The  approach  generalizes  a straightforward 
fashion  to  optimum  vector  quantization  and  its  application  to  speech 
compression  was  described  above.  This  work  has  been  written  up  and 
submitted  for  publication  [8]. 
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SECTION  II  - D 


PROOF  OF  OPTIMALITY  OF  TREE  CODES 

s 


The  convolutional  source  encoding  technique  described  earlier  in 
the  section  used  a convolutional  code  at  a rate  R > 1 for  obtaining 
compression.  Previously,  Goblick  had  suggested ’the  use  of  normal 
convolutional  codes  with  R < 1 for  source  coding.  *ln  order  to  obtain 
compression,  he  reversed  the  roles  of  the  encoder  and  decoder.  Jelinek 
gave  a proof  of  the  optimality  of  such  codes,  but  we  found  an  error  in 
this  proof  which  invalidates  it  for  all  but  symmdtri?  sources.  Utiliz- 
ing recent  results  from  the  theory  of  branching  processes  with  random 
environments  we  re-established  the  theorem  for  #te  entire  class  of 
discrete  memoryless  sources.  The  new  proof  also  indicates  that  certain 
changes  in  the  encoding  algorithm  are  necessary  if  the  complexity  is  to 
remain  within  reason.  A paper,  "On  Tree  Coding  With  a Fidelity  Criterion," 
has  been  published  in  the  IEEE  Transactions  on  Information  Theory 
(July  1975). 
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SECTION  II  - C 


MULTIUSER  COMMUNICATIONS 


Until  recently,  most  coding  techniques  were  designed  for  point- 
to-point  communication  over  a single  transmission  link.  Recently  it 
has  been  found  that  optimizing  the  coding  for  an  entire  communication 
network,  with  more  than  one  transmitter-receiver  pair,  can  yield 
superior  performance  to  that  obtainable  by  optimizing  each  link  separate- 
ly. Initially,  all  of  these  analyses  were  either  of  the  broadcast 
channel  with  one  transmitter  and  several  receivers,  or  of  the  multi- 
access channel  with  several  transmitters  and  one  receiver.  We  obtained 
some  of  the  first  results  for  multiuser  networks  with  more  than  one 
transmitter  and  more  than  one  receiver.  The  network  studied  was  the 
simplest  of  this  type,  with  two  transmitters  and  two  receivers,  and  with 
additive  Gaussian  noise.  We  found  lower  bounds  on  the  capacity  region 
for  this  channel  which  we  believe  to  be  tight.  However,  we  have  not 
been  able  to  obtain  comparable  upper  bounds.  The  few  upper  bounds  we 
have  are  quite  loose  over  most  of  their  range.  Even  so,  we  have  been 
able  to  establish  several  interesting  properties  of  this  multiuser 
system.  Consider  the  situation  in  which  transmitter  1 wishes  to  communi- 
cate with  receiver  1,  and  transmitter  2 wishes  to  communicate  with 
receiver  2,  but  there  is  interference  between  their  signals.  In  the 
case  of  symmetrical  interference  we  have 


r-j  (t)  = s j ( t ) + n/oi  s2(t)  + n-j  (t) 

r2(t)  = s2(t)+^s,(t)+n2(t) 
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w 1 . 1 — r- — 11 1,1  - 

' ' '^1 

i 

where  r^(t)  is  the  received  waveform  at  receiver  i,  s^(t)  is  the 
waveform  transmitted  by  transmitter  i,  a is  the  interference  coefficient 
(in  power),  and  n^(t)  is  white  Gaussian  noise  introduced  at  receiver  i. 

We  further  symmetrize  the  problem  by  restricting  each  transmitted  signal 
to  have  power  P,  and  by  making  both  noises  of  the  same  spectral  height 
Nq/2.  If  a = 0 there  is  no  interference,  and  each  transmitter  can  send 
at  full  capacity 

C = W log2(l  + (P/N0W))  bps 

where  W is  the  available  bandwidth.  As  a is  increased  it  is  not 
possible  for  both  transmitters  to  operate  at  capacity.  Time  division 
use  of  the  band  would  allow  each  transmitter  to  send  at  C/2  under  a peak 
power  constraint.  Frequency  division  would  perform  somewhat  better,  but 
. still  well  below  C for  usual  values  of  SNR. 

Surprisingly,  if  a is  large  enough  and  each  transmitter  uses  all 
of  the  bandwidth  all  of  the  time,  it  is  possible  for  each  to  transmit 
at  rate  C reliably.  Receiver  i first  detects  signal  \/a  s.(t)  , j f i. 

J 

Since  a » 1 it  can  do  this  reliably  even  in  the  presence  of  noise 
n.j(t)  and  "interference"  s^(t),  which  is  really  the  desired  signal. 

Once  \<a  s.(t)  is  known,  it  is  subtracted  from  r.(t)  to  obtain 

J * 

s.(t)  + n.j(t),  which  corresponds  to  no  interference  and  therefore  allows 
full  capacity  to  be  used! 

The  above  argument  is  somewhat  limited  in  direct  practical  value 
since  a is  not  usually  known.  However,  there  are  techniques,  such  as 
noise  cancelling,  which  work  well  when  there  is  strong  interference  and 
the  above  argument  adds  to  their  theoretical  basis. 
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A paper  on  this  topic,  "A  Case  Where  Interference  Does  Not  Reduce 
Capacity",  has  appeared  in  the  IEEE  Transactions  on  Information  Theory 


(Sept.  1975). 

Another  contribution  in  the  multiuser  area  is  a paper,  "On  Wyner's 
wiretap  channel,"  which  appeared  in  the  IEEE  Transactions  on  Information 
Theory  (May  1977).  This  paper  shows  that  it  is  possible  for  the 
legitimate  transmitter  and  receiver  to  communicate  at  full  capacity  and 
to  keep  the  "wire  tapper"  completely  ignorant  of  a large  portion  of  the 
message.  This  is  an  improvement  over  the  previously  known  results  which 
required  a loss  of  capacity  to  achieve  complete  secrecy. 

We  also  showed  that  feedback  can  be  used  to  advantage  on  both  wire- 
tap and  multiple  access  channels. 

Also  in  the  area  of  multiuser  communication  systems,  a paper, 
"Bistable  Behavior  of  ALOHA-Type  Systems,"  appeared  in  the  IEEE  Trans- 
actions on  Communications  (April  1975).  This  paper  describes  a poten- 
tial instability  in  packetized  radio  networks  and  suggests  mehtods  to 
remove  the  instability.  ARPA  is  currently  testing  the  packet  radio  con- 
cept for  military  use,  and  preliminary  indications  are  that  it  is  at- 
tractive for  many  applications.  The  importance  of  avoiding  instabili- 
ties is  increased  because  they  tend  to  occur  just  when  the  system  is 
most  needed  (i.e.,  when  traffic  increases). 
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